Efficient hybrid parallelisation of tiled algorithms on SMP clusters

This article emphasizes on load balancing issues associated with hybrid programming models for the parallelization of tiled algorithms onto SMP clusters. Although tiled algorithms usually account for relatively regular computation and communication patterns, their hybrid parallelization often suffers from intrinsic load imbalance between threads. This observation mainly reflects the fact that most existing message passing libraries generally provide limited multi-threading support, thus allowing only the master thread to perform inter-node message passing communication. In order to mitigate this effect, we propose a generic method for the application of load balancing on the coarse-grain hybrid model for the appropriate distribution of the computational load to the working threads. We adopt both a static, as well as a dynamic load balancing approach, and implement three alternative balancing variations. All implementations are experimentally evaluated against kernel benchmarks, in order to demonstrate the potential of such load balancing schemes for the extraction of maximum performance out of hybrid parallel programs.